Skip to content

[SPARK-58553][PS][FOLLOWUP] Match fmax/fmin signed-zero tie-break to installed NumPy - #57978

Closed
Spenserrrr wants to merge 1 commit into
apache:masterfrom
Spenserrrr:numpy-fmax-fmin-version-gate
Closed

[SPARK-58553][PS][FOLLOWUP] Match fmax/fmin signed-zero tie-break to installed NumPy#57978
Spenserrrr wants to merge 1 commit into
apache:masterfrom
Spenserrrr:numpy-fmax-fmin-version-gate

Conversation

@Spenserrrr

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

SPARK-58553 replaced the pandas_udf-based np.fmax / np.fmin implementations in the pandas API on Spark with native F.greatest / F.least expressions. When the two operands are equal (for example +0.0 and -0.0), the native mapping breaks the tie by returning the first operand.

NumPy changed this signed-zero tie-break at 2.3.0: >= 2.3.0 returns the first operand, while older versions return the second. The native mapping therefore matches NumPy >= 2.3.0 but disagrees with older versions on the sign of a ±0.0 result.

This PR selects the tie operand based on the installed NumPy version so the result matches np.fmax / np.fmin on that NumPy: return the first operand on >= 2.3.0, the second on older versions. The implementation stays fully native (F.greatest / F.least); only which operand is returned on a signed-zero tie differs by version.

Why not restore the original pandas_udf fallback for old NumPy? The original UDF matched the installed NumPy automatically (it calls np.fmax in a Python worker), so restoring it for < 2.3.0 would also be correct. But that reintroduces the per-batch JVM <-> Python round trip that SPARK-58553 removed, losing the performance and optimizer benefits for those users. Since the signed-zero tie is the only cross-version difference (verified exhaustively over every combination of {-inf, -2, -1, -0.0, +0.0, 1, 2, inf, nan} from NumPy 1.23.2 through 2.4.1 — the tie-break flips at 2.3.0 and nothing else changes), and NumPy < 2.3.0 is a frozen release range, selecting the matching tie operand keeps the native fast path while producing results identical to np.fmax / np.fmin.

Scope: fmax / fmin is the only affected function. The full test_numpy_compat.py suite (18 tests, including the generic mapping sweeps and every other SPARK-58532 conversion — fmod, ldexp, heaviside, reciprocal, float_power, bitwise shifts, signbit, etc.) passes on the minimum dependencies; the signed-zero tie in fmax / fmin is the only version-sensitive behavior.

Why are the changes needed?

The scheduled "Build / Python-only (Minimum dependencies of PySpark)" build (NumPy 1.23.2) fails pyspark.pandas.tests.test_numpy_compat NumPyCompatTests.test_np_fmax_fmin. The test asserts the sign bit of the result via np.signbit, and on the two ±0.0 tie rows the native mapping (first operand) disagrees with the reference computed from the installed NumPy (second operand on 1.23.2). Regular CI runs a newer NumPy (>= 2.3.0), where the native choice matches, which is why the original change passed pre-merge CI and the failure only surfaced in the minimum-dependency build.

Does this PR introduce any user-facing change?

No. There is no change relative to any released Spark version (the released implementation used the pandas_udf, which already matched the installed NumPy). This aligns the unreleased native implementation from SPARK-58553 with np.fmax / np.fmin on NumPy < 2.3.0. The numeric value is unchanged in all cases (+0.0 and -0.0 are numerically equal); only the sign bit of a zero result on a ±0.0 tie is corrected to match the installed NumPy.

How was this patch tested?

  • pyspark.pandas.tests.test_numpy_compat.NumPyCompatTests.test_np_fmax_fmin and the full NumPyCompatTests suite (18 tests) pass on NumPy 2.4.1 and in a minimum-dependency environment (NumPy 1.23.2, pandas 2.2.0, pyarrow 18.0.0) that reproduces the scheduled build.
  • Confirmed the failure reproduces on NumPy 1.23.2 without this change and is resolved with it.
  • Verified across installed NumPy wheels (1.23.2 through 2.4.1) that the signed-zero ±0.0 tie is the only fmax / fmin behavior that differs between versions, and that the tie-break flips at exactly 2.3.0.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

…installed NumPy

NumPy 2.3.0 changed how fmax/fmin break a signed-zero tie: for equal operands
(for example +0.0 and -0.0) it returns the first operand, while older versions
returned the second. The native mapping added in SPARK-58553 always returns the
first operand, which matches NumPy >= 2.3.0 but not older versions, so
test_np_fmax_fmin fails in the scheduled "Python-only (Minimum dependencies of
PySpark)" build on NumPy 1.23.2.

Select the signed-zero tie operand based on the installed NumPy version so the
result agrees with np.fmax / np.fmin. The implementation stays native; only the
tie operand differs by version.

Co-authored-by: Isaac
@Spenserrrr

Copy link
Copy Markdown
Contributor Author

Hi @zhengruifeng! This is a PR to fix min-deps build failure on SPARK-58553. Could you take a look when you have time? Thanks! cc @Yicong-Huang

@Spenserrrr
Spenserrrr marked this pull request as ready for review August 12, 2026 22:45
zhengruifeng pushed a commit that referenced this pull request Aug 13, 2026
…installed NumPy

### What changes were proposed in this pull request?

SPARK-58553 replaced the `pandas_udf`-based `np.fmax` / `np.fmin` implementations in the pandas API on Spark with native `F.greatest` / `F.least` expressions. When the two operands are equal (for example `+0.0` and `-0.0`), the native mapping breaks the tie by returning the **first** operand.

NumPy changed this signed-zero tie-break at **2.3.0**: `>= 2.3.0` returns the first operand, while older versions return the **second**. The native mapping therefore matches NumPy `>= 2.3.0` but disagrees with older versions on the sign of a `±0.0` result.

This PR selects the tie operand based on the installed NumPy version so the result matches `np.fmax` / `np.fmin` on that NumPy: return the first operand on `>= 2.3.0`, the second on older versions. The implementation stays fully native (`F.greatest` / `F.least`); only which operand is returned on a signed-zero tie differs by version.

**Why not restore the original `pandas_udf` fallback for old NumPy?** The original UDF matched the installed NumPy automatically (it calls `np.fmax` in a Python worker), so restoring it for `< 2.3.0` would also be correct. But that reintroduces the per-batch JVM <-> Python round trip that SPARK-58553 removed, losing the performance and optimizer benefits for those users. Since the signed-zero tie is the **only** cross-version difference (verified exhaustively over every combination of `{-inf, -2, -1, -0.0, +0.0, 1, 2, inf, nan}` from NumPy 1.23.2 through 2.4.1 — the tie-break flips at 2.3.0 and nothing else changes), and NumPy `< 2.3.0` is a frozen release range, selecting the matching tie operand keeps the native fast path while producing results identical to `np.fmax` / `np.fmin`.

**Scope:** `fmax` / `fmin` is the only affected function. The full `test_numpy_compat.py` suite (18 tests, including the generic mapping sweeps and every other SPARK-58532 conversion — `fmod`, `ldexp`, `heaviside`, `reciprocal`, `float_power`, bitwise shifts, `signbit`, etc.) passes on the minimum dependencies; the signed-zero tie in `fmax` / `fmin` is the only version-sensitive behavior.

### Why are the changes needed?

The scheduled "Build / Python-only (Minimum dependencies of PySpark)" build (NumPy 1.23.2) fails `pyspark.pandas.tests.test_numpy_compat NumPyCompatTests.test_np_fmax_fmin`. The test asserts the sign bit of the result via `np.signbit`, and on the two `±0.0` tie rows the native mapping (first operand) disagrees with the reference computed from the installed NumPy (second operand on 1.23.2). Regular CI runs a newer NumPy (`>= 2.3.0`), where the native choice matches, which is why the original change passed pre-merge CI and the failure only surfaced in the minimum-dependency build.

### Does this PR introduce _any_ user-facing change?

No. There is no change relative to any released Spark version (the released implementation used the `pandas_udf`, which already matched the installed NumPy). This aligns the unreleased native implementation from SPARK-58553 with `np.fmax` / `np.fmin` on NumPy `< 2.3.0`. The numeric value is unchanged in all cases (`+0.0` and `-0.0` are numerically equal); only the sign bit of a zero result on a `±0.0` tie is corrected to match the installed NumPy.

### How was this patch tested?

- `pyspark.pandas.tests.test_numpy_compat.NumPyCompatTests.test_np_fmax_fmin` and the full `NumPyCompatTests` suite (18 tests) pass on NumPy 2.4.1 and in a minimum-dependency environment (NumPy 1.23.2, pandas 2.2.0, pyarrow 18.0.0) that reproduces the scheduled build.
- Confirmed the failure reproduces on NumPy 1.23.2 without this change and is resolved with it.
- Verified across installed NumPy wheels (1.23.2 through 2.4.1) that the signed-zero `±0.0` tie is the only `fmax` / `fmin` behavior that differs between versions, and that the tie-break flips at exactly 2.3.0.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

Closes #57978 from Spenserrrr/numpy-fmax-fmin-version-gate.

Authored-by: Spenser Sun <hsun112358@gmail.com>
Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
(cherry picked from commit 614deed)
Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
@zhengruifeng

Copy link
Copy Markdown
Contributor

Merge Summary:

Posted by merge_spark_pr.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants